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Method for automatic alignment of tilt series in an electron microscope. 

The invention relates to a method for automatic alignment of tilt series in an 
electron microscope, comprising: 

* applying markers to a sample to be imaged by the electron microscope; 

* providing a tilt series of images of the sample; 

* identifying a first set of candidate markers in each of the images in the tilt series, 
and 

* attributing at least one probability parameter to each candidate marker in each 
image. 

In electron microscopy it may be desirable to obtain three-dimensional 
information on a sample to be studied in the microscope. Such information can be 
derived firom a tilt series of the sample and a subsequent reconstruction of the three- 
dimensional structure by means of a computer algorithm. In a Transmission Electron 
Microscope (TEM) the sample may have a thickness of a few hundred nanometers, the 
resolution? of the microscope being a few nanometers or even less. A tilt series is a series 
of images of one sample in which individual images are recorded while the sample is 
irradiated at various angles of incidence of the electron beam exposing the sample, e.g. 
a series of 141 images in a tilt interval from -70° to +70"* in steps of P. For a proper 
reconstruction of the structure in the volume of the sample, in particular at high 
resolutions, it is important that the measurement geometry be known; therefore it is 
important that the images be properly aligned. Lack of alignment of the individual 
pictures may occur, for example, due to temperature drift of the microscope during data 
acquisition, sample shrinkage or due to mechanical imperfections of the sample stage 
and the tilt mechanism, as a result of which the stage position cannot be determined 
with nanometer accuracy. 

In a known method for alignment of images in tilt series, marker particles, e.g. 
gold particles having a size of typically a few tens of nanometers, are applied to the 
sample. Each marker particle appears in each image of the tilt series, thus offering a 
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position reference for each image. Because of the great number of images in a tilt series 
these markers should preferably be recognized and identified automatically. Algorithms 
for automatically recognizing markers in an image have the ability to indicate a 
structure in an image as a marker without providing surety that such structure is indeed 
5 a marker. For this reason the indicated structure is labelled as a candidate marker, with 
some probability that such labelled structure is indeed a marker. 

A method for identifying a set of candidate markers in an image and for 
attributing at least one probability parameter to each candidate marker in the image is 
known from an article entitled "Scale-Space Signatures for the Detection of Clustered 

10 Microcalcifications in Digital Mammograms", IEEE Transactions on Medical Imaging, 
Vol. 18, No. 9, September 1999, pp.774-786. This article describes the way m which 
candidate markers are found in an image: see, for example, section III, subsection C of 
the article. Also, how to attribute a reliability parameter may be derived from this 
article: see, for example, section inC, Figure 9 and the corresponding description on 

15 page 779 left column, first fiill section, in which the size and the contrast are used to 
decide whether or not an indicated structure is recognized. No further indications for 
recognizing markers can be derived from this article. 

It is an object of the invention to provide a method for automatically aligning tilt 
series in an electron microscope by recognizing markers in the sample to be studied. 

20 According to the invention this object may be satisfied in that the method further 
comprises: 

* selecting a second set as a subset of candidate markers from the first set of 
candidate markers on the basis of said at least one probability parameter; 

* projecting the candidate markers in the second set onto a sole image; 

25 * applying a fitting algorithm to determine a set of parallel straight lines or very 
elongate ellipses best fitting the candidate markers in the sole image; 

* aligning the images in the tilt series on the basis of the identified candidate 
markers. 
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In the method of finding the markers in the images of the tilt series one may start 
by identifying candidate markers in each of the images in the tilt series in a rough 
manner. When applying this rough selection method, many candidate markers that are 
not real markers (false positive candidates) will be part of its selection result. By 
5 applying the selection step on the basis of the probability parameter according to the 
invention, a lot of false positive candidates can be deleted, thus providing a higher 
amount of reliability in the second set of candidate markers, the second set being a 
subset of the first set of candidate markers. By projecting the candidate markers in the 
second set onto a sole image, the real markers will be situated on a set of parallel 
10 straight lines, the direction of these straight lines being perpendicular to the rotation axis 
used as the tilt axis in obtaining the tilt series of images of the sample. The other 
candidate markers (i.e. the remaining false positive ones) will be scattered in a more or 
less arbitrary way over the sole image obtained by the projection step. By applying a 
proper fitting algorithm the parallel straight lines or very elongate ellipses fitting the 
1 5 candidate markers in the sole image can be determined, preferably by using a computer. 
Sometimes it may happen that very elongate ellipses, instead of real straight lines, will 
occur in the sole image. This artefact is caused by the tilt axis not being exactly 
perpendicular to the optical axis of the electron beam. After having determined the set 
of straight lines, the direction of the tilt axis is known and the true markers (i.e. the ones 
20 that are situated on the set of straight lines) can be identified as such and subsequently 
used for aligning the images in the tilt series. This alignment may then be carried out in 
a manner known per se. 

In a preferred embodiment of the invention the fitting algorithm used to 
determine the set of parallel straight lines comprises the Hough transformation. In 
25 particular in the use of a computerised recognition process for recognizing straight lines 
in the multitude of candidate marker points, the Hough transformation (known per se) 
has proven to be a reliable and stable algorithm for this recognition process. 

In another embodiment of the invention the fitting algorithm used to determine 
the set of parallel straight lines or to determine a set of very elongate ellipses is 
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constituted by the Generalized Hough transformation. It may happen that candidate 
markers belonging to one straight line are scattered in such a way that they are not 
aligned according to a straight line but — due to some undesirable influences — they 
are approximately arranged according to very elongated ellipses. In such circumstances 
5 the Generalized Hough transformation (known per se) is a good algorithm to find the 
directions of the long axes of the ellipses best fitting the candidate markers so arranged. 

In still another embodiment of the invention a cross correlation process is 
applied to the images of the tilt series before identifying candidate markers in each of 
the images in the tilt series. One may start with a rough mutual alignment of the 

10 individual images of the tilt series, which has the effect that the projections of the 
markers onto the sole image are approximately situated on straight lines rather than 
being scattered more or less at random over the sole image; this rough first alignment 
step provides a concentration of the markers on the straight lines, and has the advantage 
that the set of straight lines found by the fitting algorithm will contain very few false 

15 positive ones. This rough first alignment step may be constituted by said cross 
correlation process. 

In still another embodiment of the invention the probability parameter is derived 
fi-om at least one of the quantities: size of the marker and local contrast of the marker. 
These quantities can be derived relatively easily by means of a computerized analysis of 
20 the images, and they have proved to be reliable quantities for determining — in a first 
selection process — whether an initially indicated structure might be identified as a 
candidate marker. 

In an embodiment of the invention comprising the Hough transformation as the 
fitting algorithm used to determine the set of parallel straight lines, the fitting algorithm 
25 further comprises: 

* deriving for each candidate marker a sine-shaped curve based on the co- 
ordinates of the corresponding candidate marker, according to the Hough 
transformation; 
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* deriving from the sine-shaped curves a number of histograms indicating, for 
each direction, the relation between the density of candidate markers and the line 
distance parameter according to the Hough transformation; 

* applying an entropy operation to each of the histograms, resulting in a set of 
entropy parameters, one entropy parameter for each histogram; 

* establishing the minimum value in the set of entropy parameters; 

* identifying the histogram corresponding to said minimum value as the one 
showing the highest degree of peak diversity; 

* selecting from the latter histogram a number of peaks; 

* deriving from each peak position in the histogram the corresponding line 
distance parameter according to the Hough transformation. 

The above set of technical measures describes in more detail the way in which 
the parameters identifying the set of straight hnes containing the true marker positions 
are determined. Such process can be executed in a convenient way by using a computer. 

The invention will be described in more detail hereinafter with reference to the 
Figures, in which identical reference numerals denote corresponding elements, and 
wherein: 

Figure la : is a schematic view of a sample being exposed during a tilt series; 
Figiire lb : is a schematic image in which the markers of a tilt series are 
projected; 

Figure 2a : is an image of an object in which all candidate marker points are 
indicated; 

Figure 2b : is an image of an object in which a rough selection of marker points 
has been made; 

Figure 2c : represents the projection of the selected candidate markers of the tilt 

series; 

Figure 3a : is a graphical representation of a straight line for the purpose of 
explaining the application of the Hough transformation according to the invention; 
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Figure 3b : is a is graphical representation of the Hough transformation of a 
bundle of straight lines for the purpose of explaining the application of the Hough 
transformation according to the invention; 

Figure 4 : is a histogram representing the number of markers as seen in a first 
5 direction; 

Figure 5a : is an auxiliary figure for the purpose of explaining the selection of 
the line direction corresponding to the tilt axis; 

Figure 5b : is an auxiliary figure representing the peakiness of the histograms 
according to figure 4, for selecting the line direction corresponding to the tilt axis; 
10 Figure 6 : is a histogram corresponding to the line direction of the tilt axis. 

Figure 1 shows a schematic view of a sample 2 being exposed by an electron 
beam 4 during a tilt series. The sample may have a thickness typically of the order of 
magnitude of a few himdred nanometers. During a tilt series the sample is rotated about 

15 a tilt axis 6, which in this Figure is perpendicular to the plane of drawing. The rotation 
takes place through some fixed angle, e.g. firom -70"^ to +70'* with steps of 1°, which 
means that, in such a tilt series, 141 exposures are made. The various orientations of the 
sample during the tiU series are indicated as 2-1, 2-2, 2-3 etc. The sample is provided in 
a known way with marker particles (markers) in the form of gold particles having a size 

20 of typically a few tens of nanometers. This can be done by immersing the sample in a 
fluid provided with a gold solution, which fluids are commercially available. After 
immersion of the sample in the fluid, a number of gold particles remain on the sample in 
such a way that they can serve as markers. These markers are indicated in the figure by 
reference numerals 8-1, 8-2 etc. It should be remarked that, in general, the direction of 

25 the tilt axis 6 with respect to the image of the sample is not known, e.g. because of the 
image rotation caused by the use of magnetic electron lenses. During rotation of the 
sample in a tilt series, the alignment of the sample is not known exactly, for various 
reasons such as temperature drift of the sample stage during the series, or play in the 
moving parts of the sample stage. In order to obtain a good spatial reconstruction of the 
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image of the sample, aligmnent of the images of the tilt series must be possible, for 
which purpose the markers serve as reference points for the alignment. Therefore, any 
particular marker in a given image must be recognized in the other images. The 
invention offers an easy and reliable method of recognizing the markers in the 
5 individual images. 

Figure lb illustrates the principle as to how the markers contribute to projected 
lines, such that individual markers may be recognized. The figure provides a view of the 
sample 2 as seen firom the direction of the incident electron beam. The rotation axis 6 is 
located in the plane of the drawing, as well as the image plane 2 on which all markers 

10 are projected. Some markers 8-1, 8-2, 8-3, 8-4 etc. are shown in the figure. In executing 
a tilt series the markers 8-i are turned about the axis, thereby describing part of a circle 
having a plane that is perpendicular to the axis of rotation 6. The orbits of the markers 
along the circle are projected onto the image plane 2, which is illustrated with reference 
to marker P -3. Each image i of the tilt series provides a projection 12-i on image plane 

15 2. In ideal circumstances (no drift, no play, etc.) the set of projections 12-i are arranged 
along a straight line 10-3; in real circumstances the set of projections 12-i are not 
exactly arranged along that straight line 10-3, but more or less scattered about this line. 
It is an object of the invention to identify such a straight line fitting the scattered 
projections 12-i. 

20 Figures 2a, 2b and 2c illustrate a sample to be studied by means of a tilt series. 

The sample 2 consists of an object 14, and is provided with gold particles as markers. 
These markers may be identified by applying a first (rough) selection method known 
per se. However, as a result of this first rough recognition step many candidate markers 
that are not real markers may be identified as true markers (false positive candidates). 

25 The first set of markers so obtained is represented in Figure 2a by references 8-1 , 8-2, ... 
in general 8-i. After this first recognition step a following selection step is made; in this 
latter step one or two probability parameters are attributed to each candidate marker in 
each imag:;, and a second set is selected as a subset of candidate markers fi-om the first 
set of candidate markers on the basis of said probability parameters. The result of this 
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selection step is represented in Figure 2b, in which some markers of the first set 
considered to be false positive candidates are indicated by dashes 18-i, and the markers 
of the first set that are selected as true markers (the second set of markers) are indicated 
by crosses 16-i. In Figure 2c the result of the projection of the true markers 16-i onto a 
5 single image plane 2 is illustrated. As already described with reference to Figure lb 
markers 16-i rotate about the tilt axis 6. In principle each marker will produce a 
rectilinear trace 20-i in said image plane, but some markers are positioned in such a way 
that their traces coincide, see, for example, marker pairs 16-1, 16-2 and 16-3, 16-4. The 
straight lines obtained in this way are detected by means of an algorithm comprising the 

10 Hough transformation, as will be described with reference to Figures 3a and 3b. 

In Figure 3a a straight line 22-1 is represented in a usual system of (x,y) axes. A 
point of tliis line is denoted as Xq, yo- A bundle of straight lines all going through point 
xo, yo is denoted in general as 22-i. The usual equation of such a straight line 22-1 is 
y=rx+c, in which r=tan(a), a being the angle between the line 22-1 and the positive x- 

1 5 axis, and c being the intersection of the line 22-1 with the y-axis. As is generally known, 
another representation of the line 22-1 is expressed as r=y cos(a) - x sin(a) (r being the 
distance firom the origin x=0,y=0 to line 22-1), in which x and y represent points on the 
line 22-1 . In this way any line containing point xq, yo is expressed as r=yo cos(a) - 
xo sin(a), each set of values of the pair (r,a) now indicating a line of the bundle 22-i, 

20 The latter expression may be represented by one point in a set of (a,r) axes: see Figure 
3b. So line 22-1, having one particular value of r and of a, is represented by one point 
22-1 in Figure 3b. In the same way all other lines going through point xo, yo in Figure 3a 
are represented by a point in Figure 3b, all lines 22-i in Figure 3a providing points 22-i 
in Figure 3b, and all points 22-i together constituting a sine-shaped line 24. 

25 A bundle of lines all going through another point xi, yi in Figure 3a (not drawn) 

will give rise to another sine-shaped line in Figure 3b; if these two sine curves intersect, 
it means that the two points xo, yo and xi, yi are situated on the same line in Figure 3a, 
which is always true for two points. Finding points that are all situated on the same line 
means — in terms of the Hough representation of Figure 3b — that one should look for 
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intersection points of all sine curves corresponding to those points. If these sine curves 
all have the same intersection point, it means that all points Xi, yj are situated on the 
same line in Figure 3 a. 

Applying the above described algorithm to the projected marker points of Figure 
5 2c (which are not individually drawn but are represented by lines 20-i) means that for 
each candidate marker point the corresponding sine-shaped curve should be established. 
So, as an example, if there are 50 marker points identified as being true markers (the 
second set of markers) and the tilt series consists of 141 images (tilt interval from -70® 
to +70"^ in steps of 1*) then there are 50x141=7050 points in Figure 2c that give rise to a 
10 sine curve as in Figure 3b. To find lines in Figure 2b containing a high density of 
marker points one should find all intersection points of all sine curves according to 
Figure 3b. This means that for n sine curves one should calculate ?4n(n-l) intersection 
points to be represented in Figure 3b. 

This calculation of all intersection points will provide the areas having a high density of 
1 5 intersection points in Figure 3b; a high density of intersection points in Figure 3b 

corresponds to lines containing relatively many marker points according to Figure 2b. 

For a numerical analysis of the density of intersection points in the (a,r) 
representation the following operation is carried out. The complete interval in a is 
subdivided into (equal) small intervals Aa around each value of a (that particular value 
20 of a being denoted as Oo). For each value of Oo a histogram is formed, which means that 
the distribution of the number of markers for all values of r is determined. The result of 
this operation is a collection of histograms equal in number to the number of intervals 
into which a is subdivided. One such histogram is represented in Figure 4. It should be 
noted that the distribution of the numbers of markers over the various values of r is not 
25 very divergent, which means that translating a small window (width Ar) at a value of Oq 
in the direction of r does not result in a large variation in the numbers of markers that 
appear together in the window. A graphic representation of the latter operation is 
illustrated in Figure 5a. Figure 5a shows the x,y system in which all (projections of) 
markers are placed. Finding the directions in which concentration lines of markers are 
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present involves defining a window having width Ar and some direction a. Two 
windows 26 and 28 are drawn, window 26 having direction ai and window 28 having 
direction aa. It can be seen from Figure 5a that the direction of concentration lines of 
markers coincides with direction ai. When window 28 is translated across the x,y 
system, one observes lhat there are no areas in that x,y system exhibiting concentrations 
of markers, so the histogram formed by this operation will not show a high degree of 
variation in the numbers of markers observed in that window, i.e. at the various values 
of r. The same holds for most other possible directions of such a window. However, 
translating window 26 having direction ai results in the passage of the window over the 
concentration lines having the same direction ai, which will result in relatively many 
markers being detected in that window at the corresponding value of r, alternated by 
areas in which relatively few markers appear in the window. Such a distribution is 
illustrated in Figure 6, to be discussed later with reference to that Figure. 

The above graphical operation for finding the concentration lines of markers will 
now be described in a mathematical way suitable for application in a computer. For all 
values of a, a histogram r,n (i.e. the number of markers n versus the distance parameter 
r) 

is formed. Each histogram is subjected to an entropy calculation according to the 
following expression: 




in which expression S is the entropy to be calculated, nj is the number of markers 
detected at some value of the distance N is the total number of markers in the 
histogram for which S is to be calculated, M is the number of intervals Aa and "In" 
means the natural logarithm, which is the logarithm to the base e, where e is 
approximately equal to 2.718... The quantity S is, as is generally known, a measure of 
the divergence of the distribution of the numbers of markers over the various values of 
r; in other words S is a measure of the "peakiness" of the histogram. Now all histograms 
are subjected to a calculation to obtain the value of S according to the above expression, 



FNL0202US 



11 9.12.2002 

and this range of values is represented as a function of the direction a in Figure 5b. It 
should be noted that a low value of S in this Figure 5b means that the histogram 
belonging to it has a high "peakiness". In Figure 5b the minimum value of S 
(corresponding to the maximum "peakiness") occurs at a=77®, so that one may assume 
this to indicate the value of the direction perpendicular to the tilt axis with respect to the 
chosen system of (x,y) axes. 

Figure 6 shows the histogram corresponding to the extremal value of S referred 
to above, i.e. the line direction perpendicular to the tilt axis. Comparing the histogram 
according to Figure 6 with the histogram of Figure 4 it is clear that the first histogram is 
much more "peaky", and thus has a very high probability of being the one that 
corresponds to the direction perpendicular to the tilt axis. From Figure 6 one may 
identify the various parallel lines of markers as schematically depicted in Figure lb, 2c 
or 5a. Once the various lines have been identified it is easy to identify the various 
positions of the projection of the single markers that gave rise to the relevant line (all 
other candidate markers being classified as false positive ones) , and based thereon to 
mutually align the individual images of the tilt series. Such alignment is known per se to 
the person skilled in the art. 



